In the case of obtaining (capturing) an image by a digital camera, noise may sometimes be added to the image due to the characteristics of a CCD (Charge-Coupled Device) or a readout circuit for CMOS or the characteristics of transmission paths. Also, blur of an image due to an out-of-focus condition in capturing of the image or blur of an image due to camera shake occurs. In this way, the captured image has blur that is attributed to user's handling of the camera in photographing in addition to the noise that is attributed to the specific characteristics of the captured image, so that the image is degraded. Among such “blur” types, blur of an image that is attributed to a motion of a camera during photographing (exposure) is sometimes referred to as “motion blur”.
In recent years, especially, demand for high sensitive photography is increasing, and therefore, it is necessary to restore an image degraded by blur (hereinafter, “degraded image”) to an image that is as close to an original image (hereinafter, “ideal image”) as possible. To realize a bright image that is free from noise or blur, such as an image demanded in high sensitive photography, the solutions are generally divided into two ideas, increasing the sensitivity and extending the exposure time.
However, increasing the sensitivity disadvantageously leads to amplification of noise. As a result, a signal is buried in the noise so that, in many cases, large part of a resultant image is formed by the noise. On the other hand, extending the exposure time enables accumulation of a larger amount of light that occurs at that site, resulting in an image that includes smaller noise. In this case, a signal would not be buried in the noise, but there is a problem of generation of motion blur in an image due to camera shake.
According to the prior art, there are two countermeasures against the problem resulting from the extended exposure time. One is optical camera shake compensation, such as lens shift, sensor shift, or the like. The other one is obtaining the direction/magnitude of motion blur from a resultant image or through a sensor and performing signal processing based on the obtained direction/magnitude of the blur to restore the image (a restoration method based on signal processing). The restoration method based on signal processing is, for example, disclosed in Patent Document 1, Patent Document 2, and Non-patent Document 1.
Hereinafter, the restoration method based on signal processing is described.
Here, the brightness distribution of an image formed on an imaging area of the image capture device is expressed by I(x,y). The coordinates (x,y) are two-dimensional coordinates that are indicative of the position of a pixel (photosensitive cell) on the imaging area. In the case of an image formed by, for example, M×N pixels that are arranged in rows and columns, assuming that x and y are integers that satisfy the relationships of 0≦x≦M-1 and 0≦y≦N-1, respectively, the position of each one of the pixels that form the image can be specified by the coordinates (x,y). Here, it is assumed that the origin of the coordinate system, (0,0), is at the left upper corner of the image. The x axis extends in a vertical direction. The y axis extends in a horizontal direction. Note that the arrangement of the coordinates is arbitrary.
Where the brightness distribution of an unblurred image (ideal image or original image) is L(x,y), the PSF (Point Spread Function) that defines the blur is PSF(x,y), and noise is n(x,y), Formula 1 shown below holds:
[Expression 1]I(x,y)=PSF(x,y)*L(x,y)+n(x,y)  (Formula 1)Here, symbol “*” means two-dimensional convolution.
The point spread function PSF(x,y) of camera shake depends on the trajectory of the camera shake during photographing (exposure). The trajectory of the camera shake varies among shots of photography with a camera, so that PSF(x,y) also varies among shots of photography with a camera.
The trajectory of the camera shake during photographing may be detected by a gyro sensor, or the like. When PSF(x,y) is known, deconvolution is performed using this PSF(x,y), whereby the image L(x,y) can be restored based on the degraded image I(x,y). On the other hand, when PSF(x,y) is unknown, it is necessary to estimate PSF(x,y) from the degraded image I(x,y) for restoration of the image L(x,y). The former is referred to as “non-blind deconvolution”, and the latter is referred to as “blind deconvolution”. In the case of the blind deconvolution, it is necessary to estimate both PSF(x,y) and the image L from the degraded image I. Therefore, reduction of blur is difficult as compared with the non-blind deconvolution.
The convolution of a PSF that defines blur caused by camera shake (camera shake blur) is realized by a linear filter. The linear filter of the two-dimensional convolution is typically expressed by a kernel that is formed by a coefficient matrix that has a size of N×N pixels. Here, N is an integer not less than 3. The PSF that defines the blur can be expressed by a blur kernel. To process an image that has blur into a restored image in which the blur is reduced (including an image from which the blur is eliminated), it is necessary to estimate a blur kernel that defines the blur.
Non-patent Document 1 discloses using a multi-scale inference scheme in estimating a blur kernel and a restored image from a single image that has blur. In this multi-scale inference scheme, a low-resolution degraded image is first used to estimate a blur kernel that has a size of 3×3 pixels. Then, the resolution of the degraded image used for the estimation is gradually increased, whereby the resolution of the blur kernel is increased. FIGS. 13(a) to 13(h) are diagrams for illustrating the scheme disclosed in Non-patent Document 1. The upper part of FIG. 13(a) shows a blur kernel of 3×3 pixels, and the lower part shows a restored image of a corresponding resolution. The same applies to FIGS. 13(b) to 13(h), although with gradually increasing resolutions.
The low-resolution degraded image has a small pixel count, and accordingly, the pixel size of the camera shake blur is small. As a result, in the low-resolution degraded image, the size of the blur kernel is smaller, and accordingly, the amount of computation that is necessary for estimation of the blur kernel is smaller. If estimation of the blur kernel is started with a high-resolution degraded image that has a larger pixel count, there is a probability that convergence to a blur kernel that is different from a true blur kernel (local minima) may occur. According to the multi-scale inference scheme, the estimation accuracy of the blur kernel can be improved.
In order to estimate a blur kernel K which has a size of M×M pixels (size M) with the use of a blur kernel K which has a size of N×N pixels (size N) where M is an integer greater than N, it is necessary to enlarge the blur kernel K (size N) by a factor of M/N. Also, it is necessary to prepare a degraded image obtained by enlarging the pixel size of the degraded image used in estimation of the blur kernel K (size N) by a factor of M/N in terms of either of the vertical and horizontal dimensions. Based on the thus-enlarged degraded image, the blur kernel K (size M) is estimated from that degraded image. The blur kernel K (size M) that is used as the initial value in this estimation can be obtained by enlarging the blur kernel K (size N) that has a smaller size.